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With an emphasis on the problems of control of extraneous variables and 
threats to internal and external validity, the arrangement or design of experiments is 
discussed. The purpose of experimentation in an educational institution, and the 
principles governing true experimentation (randomization, replication, and control) are 
presented, as are three pre-experimental, three true experimental, and ten 
quasi -experimental designs with the advantages, disadvantages, and appropirate 
statistical treatments for each. This discussion follows Campbell and Stanley’s 
chapter on experimental design in Gage’s "Handbook of Research on Teaching". CMC) 
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EXPERIMENTAL AND OUASI-EXPERIMENTAL DESIGN 



Experimental Design 

Exper ime ntal research design has as its purpose the establishment of causal 
relationships between variables. It implies control of independent Variables 
(experimental group -vs- control group). An attempt is made to generalise from 
experience to law/s or principles. Effective experimentation requires careful 

planning. 

Before beginning tb collect data for an experimental study, one should be 
Abie tb haVe completely written-up the study to the conclusion, stating criteria 
for measuring the dependent variable, controls, data format, hypothesis/es to be 
tested, including statistics to be used. 

Purpose of Experimentation 

As stated above, experimental research seeks quantitative formulation of 
verifiable general lavs and its ultimate aim is establishment of a system of 
concepts and relations (the so-called nomothetic net) in which all specific 
propositions are deducible from a few general principles. This is ’basic research.” 

Most research in education, according to Ebel, should be directed toward 
studies designed to yield information immediately useful in the solution of con- 
temporary educational problems. This is usually termed ’’applied research - 
collection of data (experience) that promise help in the solution of some immediate 
practical problem. Applied research can still be quite carefully designed end con- 
trolled, involving a good deal of design and statistical sophistication. 

Institutional research can be basic or applied, but most often it is applied 

research. 

Principles Governing True Experimentation 
Achievement of the three following principles is ’’something else.” 
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1* RANDOMIZATION - in which it is left to chance, or random methods, 
the determination of which subjects receive which treatment, 
there will undoubtedly be some factors over which we have 
little or no control. This is the chief principle underlying 
experimentation* 

2. REPLICATION - in which we must be able to repeat or reproduce the 

experiment with a number of subjects and in a number of dif- 
ferent settings (internal and external validity). 

3. CONTROL « in which the experimenter controls or manipulates what 

happens to whom, when and where* it is possible t ?> build 
into the study a comparison groiip which i A similar to the 
experimental group except that it did net receive experimental 

treatment. 

Since we are not always able to achieve these H ideal'* principles, we will 
later need to look at some Quasi-experiment a 1 Designs . In true experimental 
design we need to consider eight factors which may jeopardize internal , validit y 
and produce effects on our subjects confounded with the effect of the experimental 
stimulus or treatment, and four factors which may jeopardize external valid i ty ; or 
the generalizability of the findings beyond the loeal population or sample of our 

study. 
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Principles of True Experimental Design 

A. Factors which may jeopardize internal validity 

1. HISTORY * specific events beyond the experimental variable treatment 
which occur between critical measurements and over which the 
experimenter has little or no control, (eg - Between pre-test 
and post-test, in an experiment on effect of a reading acceler- 
ation machine on reading rate, and the day before retesting, a 
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major event occurs, such as a paralyzing winter storm or the 
president is assassinated) 

2* MATURATION - processes within the experimental subjects tfiich operate 
as a function of passage of time per se. (Eg - growing older, 
getting hungrier, getting more tired, etc,) 

3* TESTING EFFECT - factors which affect or threaten internal validity 
due to repetitioh of the same test, or of another form of the 
Same test* and will tend to produce some change in results simply 
due to test detest situation alone. 

4. INSTRUMENTATION EFFECT « changes in calibration of a measuring instru- 

ment OR, Hore usually, changes in the "observer's" or "rater's" 
judgment. 

5. STATISTICAL REGRESSION - when experimental subjects have been chosen 

on the basis of their extreme scores on a measurement instrument, 
invalidation of the experiment is likely to occur, since scores 
on retest of this group will tend to regress toward the mean of 
that larger group from which the extreme group was selected. 

The only instance \tfien this would not be true would be if 
the test had perfect reliability (none of this type has jje£ been 
developed!) 

6. SELECTION - biases introduced due to method of selection may seriously 

threaten Internal validity. (Eg - If ra ldomness in selection of 
experimental and control groups is adhered to, selection biases 
are usually eliminated, especially if a sample of sufficient size 
is drawn) 

7. EXPERIMENTAL MORTALITY ("the bane of all experimenters") - the differ- 

ential loss of subjects in the experimental and control groups 
during the process of the experiment. 
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8, SELECTION-MATURATION INTERACTION - selection of subjects, though random, 
may produce interaction with other factors, usually maturation, 
confounding the effect of the experimental treatment. (Eg - A 
study of the "Effect of An Experimental Discussion Program tipon 
Realism of Vocational Choice." A group of sophomores is randomly 
selected and the realism of their vocational choices at end of 
experimental program is compared with realism of vocational 
choices of a random group of freshmen. Selection of more mature 
sophomores to compare with less mature freshmen has introduced 
maturation as a threat to internal validity. 

Comparison of two randomly selected groups of sophomores, 
one of which received the experimental discussion program, would 
have been more desirable. 

B« Factors irtiich may jeopardize external validity 

9. THE REACT IVE OR INTERACTIVE EFFECT OF TESTING, AS SUCH - the act of 

pretesting experimental and control groups may affect their 
reaction or responses, increasing or decreasing subject s 
sensitivity, thus making results not generalizable to the un- 
pretested population as a whole. 

10. INTERACTION EFFECTS OF SELECTION BIASES AND THE EXPERIMENTAL VARIABLE /S - 

(Eg - The effect of a course, "History of Religions", upon tol- 
erance in a small rural school might show a different correlation 
with growth in tolerance than the same course might show in a large 
urban or suburban school.) 

11. REACTIVE EFFECTS OF EXPERIMENTAL ARRANGEMENTS - the reaction of subjects 

to knowing they are "in an experiment" and the objects of special 
attention may produce effects which confound experimental treatment 
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(The famous Hawthorne Effect is an example of this, in which 
women workers in both the experimental and control groups showed 
significant increases in production just as a result of being 
shown special attention) 

12. MULT IPLE -TREATMENT INTERFERENCE - whenever multiple treatments are 

applied to the same subjects, the effects of the prior treatments 
may not be completely erasable. (Eg - A study of the effect of 
classroom climate upon learning might subject one randomly se- 
lected group to an authoritarian climate the first 6-week period, 
another to a ,, democratic ,, climate, and a third to a "laissez- 
faire" climate. The second 6-week period, the first teacher • 
would "switch" to a democratic climate, the second to laissez- 
faire, and the third to authoritarian, and so on. The effects of 
authoritarian climate upon the first group could net. possibly be 
kept from confounding their subsequent experience with later 
democratic and laissez-faire treatments.) 

3 

Three Pre-experiental Designs 

Design 1. The One-Shot Case Study - Much research in education conforms to 

a design in which a single group is studied only once, subsequent 
to some agent or treatment presumed to cause change* Such studies 
might be diagrammed as follows: 

X Ol 

(Treatment) (Observation or measurement of effect) 

Such studies have such a total absence of control as to be 
of almost no scientific value. Careful, elaborate measurement and 
use of detailed description and statistics in such studies represent 
'•The Error of Misplaced Precision." 
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Design 2. 



Design 3. 



Design 4* 
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The One "Group Pretest-Posttest Design 

0 X o 2 

Reference to Campbell and Stanley will present the many 
weaknesses and the slight increase in internal validity of this 
design over Design 1 above. Some control is gained in the Se- 
lection and Mortality factors. It is still considered "bad 
design" for educational research. 

The Static-Group Comparison Design - This is a design in which 
group which has experienced X is compared with a group which has 
not, for she purpose of establishing the effect of X. 

X o 1 

o 2 

This design controls a few more of the factors (threats to 
internal validity), but now does not control for Selection and 
Differential Mortality between the two groups. 

4 

Three True Experimental Designs 
The Pretest-Posttest Control Group Design 
R 0^ X O2 

°3 °* 

(Infers random selection) 

This design, if carefully followed, neatly controls at least 
seven of the eight internal validity threats (See TABLE 1 in the 

paperback, p. 8). 
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Design 5. 



Design 6. 



The Solomon 4-Group Design - 

R x (Group 1, exper.) 0! X 0 2 

r 2 (Group 2, Control) 0^ ®4 

r 3 (Group 3, Exper.) x 0 5 

r 4 (Group 4, Control) °6 

This design parallels the Design 4 groups with two additional 
randomly selected groups (experimental & control) which lack the 
pretest. Therefore, both the main effects of te sti ng, and the 

interaction of testing and X are determinable. 

in this way, not only is the generalisability increased, but 

in addition, the effect of X is replicated in four different 
fashions: 0 2 > 0 X , 0 2 > 0 4 , 0 5 > 0 6 , and 0 5 > 0 3 . 

Design 5 lends itself to use of the 2 X 2 analysis of variance 

design in treating the posttest scores. 

No X X 

Pretested ®4 ^2 

Unpretested °5 

If the main and interactive effects of pretesting are negli- 
gible, it may be desirable to perform an analysis of covariance of 
0 4 versus 0 2 , pretest scores being the covariate. 

The Posttest-Only Control Group Design 



R X 0 X 
R 0 2 

•The pretest is not actually essential to true experimental 
designs," say Campbell and Stanley. They go on to say, "For 
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psychological reasons it is difficult to give up ’knowing for 
sure' that the experimental and control groups were 'equal' 
before the differential experimental treatment. Nonetheless, 
the most adequate all-purpose assurance of lack of initial 
biases between groups is randomization."^ 

Design 6 is appropriate to all of the settings in which 
Designs 4 or 5 might be used, i.e. f designs where true randomiz- 
ation is possible. A very lucid account of the advantages and 
disadvantages of Design 6 over Designs 4 and 5 is given by 
Campbell and Stanley here* 

* • i . I 

t i , 

The £ test is the simplest form of statistical treatment 
for Design 6, the only setting fdr which this test is optional. 
Covariance analysis and blocking on "subject variables'' <B. J. 
Underwood, Psychological Research . New York; Appleton - 
Century-Crof ts , 1957) such as prior grades, test scores, parental 
occupation, etc., can be used, thus providing an increase in the 
power of the significance test very similar to that provided by 
a pretest. 

Quasi-Experimental Designs 

In situations occurring in many natural social settings, where full experi- 
mental control is lacking, the researcher can still introduce something like 
experimental design, even though he cannot have full control over scheduling of 
experimental stimuli and of randomization. He still bas some measure of control 
over when and to whom measurement data can be recorded on. It becomes Imperative, 
then, that the researcher be fully aware of which specific variables his parti- 
cular design fails to control. 
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Please refer to the introductory paragraphs on quasi-experimental designs 
and the theory of experimentation for a lucid overview of the place of expert 
mentation in controlled laboratory situations and in the field. 

The inherent strengths and weaknesses of the following list of quasi 
experimental designs are given in Campbell and Stanley’s chapter of Gage’s 
Handbook . . . (and the paperback copy of this chapter). These cannot be covered 

in this brochure in brief or cogent fashion. 



Design 7. 



Design 8 . 



Design 9. 



Design 10. 



The Time-Series Experiment 

°1 °2 °3 °4 X °5 °6 °7 °8 

The Equivalent Time-Samples Design 

x x 0 x 0 0 x x 0 x 0 0 etc. 

The Equivalent Materials Design 
M a X 1 0 M b X 0 0 M^O M d X 0 0 etc. 

The Nonequivalent Control Group Design 
0 

0 0 



Design 11. Counterbalanced Designs 





Time 1 


Time 2 


Time 3 


Time 4 


Group A 


XjO 


X 2 0 


X 3 0 


X 4 O 


Group 6 


X 5!° 


x 4 o 


x t 0 


x 3 o 


Group C 


x 3 o 


X t 0 


v 


X 2 ° 


Group D 


V 


X 3 ° 


x 2 0 


XjO 



Design 12. The Separate-Sample Pretest-Posttest Design 

R 0 (X) 

R X 0 



o 

ERLC 



10 



» 



Design 13. The Separate-Sample Pretest-Posttest Control Group Design 

R 0 (X) 

R X 0 

R 0 

R 0 

Design 14. The Multiple Time-Series Design 

2 _ 2 * 2 - 2 - 5 - 2 . 2 - 2.2 

oooo oooo 

Design 15. The Recurrent Institutional Cycle ("Patehed-Up") Design 

Class A X 0 ^ 

Class B °2 X °3 



Design 16. Regression-Discontinuity Analysis 

This quasi-experimental design attempts to substitute for 

the true experimental Design 6 by examining the regression line 
between independent variable (such as ability test scores) and 
dependent variable (eg. - later achievement) for individuals in 
a narrow independent variable score interval (i.e. - with essen- 
tially equalability scores) at or slightly below the "cut off" 
score above which a certain award or motivational opportunity is 
given. If discontinuity of the two regression lines exists at 
the cut-off score, '.’...the evidence of effect would be quite 
compelling, almost as compelling as in the case of the true 
experiment."® 



Summary 

In conclusion, in this brief brochure, we have discussed alternatives in 
the arrangement or design of experiments, with particular regard to the problems 
of control of extraneous variables and threats to validity, both internal validity 
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threats (8) and external validity threats (4)* 

Sixteen experimental designs (3 pre-experimental, 3 'true , and 10 quasi 
experimental) and some variations on them have been presented. 

The check-list of validity factors (Tables 1, 2, & 3 in reference texts) 
is an important aid in interpreting the results of such experiments. 



NOTE: Almost all credit for the above must go to Campbell and Stanley's chapter 

in N # L # Cage 1 e Handbook of Research on Teaching., from which the 
"plagiarized" material was freely taken. 
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